Distributed Continuous Quality Assurance Leveraging User Resources to Improve Software Quality Around-the-World, Around-the-Clock

Quality assurance (QA) tasks, such as testing, profiling, and performance evaluation, have historically been done in-house on developergenerated workloads and regression suites. Since this approach is inadequate for many systems, tools and processes are being developed to improve software quality by increasing user participation in the QA process. A limitation of these approaches is that they focus on isolated mechanisms, but not on the coordination and control policies and tools needed to make the global QA process efficient, effective, and scalable. To address these issues, we have initiated the Skoll project, which is developing and validating novel software QA processes and tools that leverage the extensive computing resources of worldwide user communities in a distributed, continuous manner to significantly and rapidly improve software quality. This paper provides three contributions to the study of distributed continuous QA. First, it illustrates the structure and functionality of a generic around-the-world, around-the-clock QA process and describes several sophisticated tools that support this process. Second, it describes several scenarios implemented using these tools and process. Finally, it presents the results of a feasibility study applying these scenarios on two widely-used, large-scale open-source middleware toolkits. The results of this study indicate that the Skoll process and its toolsuite can effectively manage and control distributed, continuous QA processes. In a matter of hours and days we identified problems that had taken the ACE and TAO developers substantially longer to find and several of which had previously not been found. Moreover, automatic analysis of QA task results often provided developers information that quickly led them to the root cause of the